\ 



WORLD INTBLLECrUAL PROPERTY ORG AI^ZATT W 
Ihtexnaiional Bureau 




PCX 

INTERNATIONAL APPUCATTON PUBLISHED UNDER TOE PATENT COOPERATION TREATY (PCT) 



(51) Intematitmal Patent Classification ^ : 
60^ 3/00 



Al 



(11) International Fubliration Number: WO 9S/20788 

(43) International PubBcatipn Datte: 3 August 1995 (03^8.95) 



(21) IntemaUonai Application Number: PCr/yS95/01 227 

(22) International BHing Date: 27. January 1995 (27.01.95) 



(30) Priority Data: 

OS/187,880 



27 January 1994 (27,01.94) US 



(71) Applicant: EXOS. INC. [US/US]; 2A Gill Street, Wobura, MA 

01801 (US). 

(72) Inventors: OSBORNE, Timothy; 10 Udine Street, ArUiigton, 

MA 02174 (US). MARCUS, Beth. A.; 78 Wobum Street, 
Lexliigton, MA 02173 (US), 

(74) Agent: PASTERNACK, Sam; Choate, Hall & Stewart, Ex- 
change Place, 53 State Street, Boston, MA 02109 (US). 



(81) Designated States: OA, JP, MX. European patent (At, .'BE, 
CH. DE, DK, ES, PR, GB, GR. IE, IT, LU. MC NL. PT. 
SB). 



Published 

With international search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the rece^ft 
amendments. 



(54) Title: .INTELLIGENT REMOTE MULTIMODE SENSE AND DISPLAY SYSTEM UTIUZING HAPTIC tNTORMATON 
COMiPRESSIpN 



EnvlTBiunant 



10 



12 



Hiptio 
System 



Sense 



Display 




(57) Abstract • 

A system for sensing and displaying sensory information to a human operator comprismg a world modeler adapted to lecdye simsory 
input from the operator over a communications medium. The world modeler modifies eith^ a virtoal worid which Is a mathematu:al nwdel 
of an environment, or modifies a real world which is a icraoie real cnvirpnmem. TTic world modeler dien sends sensory information from 
the real or virtual environments over the communication medium to the haptic system. The haptic system displays thus sensory information 
to the human operator. iTie resuU is that the operator is able to manipulate a real or yiitual cnvironmem remotely dirou^ the mforniation 
sensed and displayed by the haptic system. The method provides for compression of the haptic infonnation,reducmg the bandwidth rcqwred 
to transmit sensory information between the haptic system and the world modeler. 
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TNTFJ JJRBN T REMOTE MULtlMODiE SENSE AND DISiPLAY 
SYSTEM UTILIZING HAPTI C INFORMATION COMPRESSION 

5 Background of the Invention 

The present invention relates generally to a system and method of 
providing haptic (position, orientation, force, & tactile) feedback to the 
human operator or a remote ;0r virtual environment. 

Devices and systems are well known for measuring pr monitoring one 
10 or more characteristics of an anatomical part (see for example U.S. Patent 
Nos. 4,986,280, 3,258,007 and 3,364,929). A human controlled position 
sensing device for use in the field of robotics is also well known (see for 
example, U.S. Patent Nos. 4,328,621, 4.534,694, 4.608,525, and 
4,674,048). 

15 

Force Feedback Systems 

Systems are also well known for providing force feedback to human 
operators. Fdrce-reflective devices exist in a variety of forms, from 
joysticks to arm and hand controllers. These devices serve as input devices 

20 and haptic interfaces for virtual environment manipulation, or teleoperation. 
A significant amount of research has been conducted on robotic hands and 
force-reflecting controllers for robotic arm manipulation. While a few 
controllers were anthropomorphic, most were typically tailored to imitate the 
slave robot's kinematics. The advantage of anthropomorphic controllers is 

25 that they imitate the kinematics of the human hand and ann, thus possessing 
the largest range of human operator (user) motion. Yet these past 
anthropomorphic designs were unable to achieve high quality force feedback 
while retaining minimal actuator size, a requirement for system portability. 
So while many forms of force-feedback devices have been developed, few 

30 completely anthropomorphic systems exist. The systems tended to be 

massive and bulky, and required large centralized computational resources 

/ 
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tightly coupled to the sensory system, further preventing an efficient 
integration of the system to the human operator. Finally, the systems were 
«non-modular, requiring extensive work to add additional haptic sense and 
display devices. 

5 

Tactile Systems 

Tactile displays have been the subject of much research activity in the 
past thirty years. While neuroscientists smdied the physiological roots of 
human tactile perception, engineers were interested in tactile display devices 
10 that substituted or augmented other senses such as sight and heariujg. Tactile 
displays also demonstrated their usefulness in virmal reality simulations and 
telerobotic operations. 

A system utilizing tactile display devices provides die teleoperator 
with additional information, such as contact locations, diat could not be 
15 adequately displayed by force alone. Tactile displays also simplify the 

requireinents placed upon force reflection. A combination of tactile display 
and force reflection is haptically more effective, technically less challenging 
and economically more feasible than force reflection alone. Therefore, a 
' high performance force reflective exoskeleton master should incorx)orate 
20 tactile displays. 

Tactile display technologies have slowly evolved in the past thirty 
years. Many different tactile display technologies are available today. For 
instance, air or water jet displays are used to trace outlines on the skin and 
are attractive because they eliminate the need for moving mechanical parts 
25 and because they exhibit a consistent frequency response. Air or water 

bladder displays are used to apply a continuous pressure on an area of skin 
but suffer from extremely low bandwidth. 

Mechanical tactof displays include most of the vibro-tactile devices 
described in the literature in addition to a few displays operated at a very 
30 low frequency that would not normally be considered vibrotacule. A few 
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Single-element display devices exiist but the vast majority of mechanical 
tactile displays involve tactpr arrays. The number of elements in these 
arrays run from two to 400. Almost all oif the devices in this group involvie 
indenting or impacting the skin perpendicular to the skin surface. Two 

5 methods of actuation dominate these displays; solenoid activated pins and 
cantilevered piezoelectric blmorph reeds with a pin mounted to the dp. 
Another method of actuation diat has great potential is the use of shape 
memory alloys. As a current is passed through the alloy, it changes shape 
and deflects a spring that holds a tactor pin in place. Voice coil displays 

10 generate vibratory sensations by mounting a voice coil against the skin. 

Electrotactile displays elicit touch sensations by indiscriminately 
stimulating all the touch and pain sensors in titie skin with a current passing 
through the skin. These devices usually involve a remote stimulator (which 
supplies the power and generates the stimulation waveform) and a number of 

15 electrodes mounted on the skin using a sticky dielectric. Each electrode pair 
constimtes a stimulation site. While some systems involve single or a small 
number of isolated electrode pairs, arrays of electrodes were also popular, 
with number of elements running up to 1024 in one instance . 

In all of these devices, the sensing or feedback element contains no 

.20 intelligence or ability to interact with human stimuli itself . They are hooked 
to a computation engine which contains a world model which interprets the 
raw sensory information and then sends low level commands for behaviors 
to be achieved. There are many limitations in this approach which have 
been apparent and much discussed in the research literamre (papers on force 

25 feedback in the presence of time delays by Sheridan and others) and 
elaborate methods pf modeling system performance to overcome these 
problems. The more complex the feedback, the more extreme the problem 
becomes. In fact this has led the virtual reality research community in the 
opposite direction finona this iiiyentipn, to use larger and more powerful 

30 coniputatipnal engine$ and very simple feedback modes with limited 



wo 95/20788 



PCT/US95/01227 



performance success.. Most of the systems used are on non real-time 
computation engines which makes the problem of providing stable, realistic 
and predictable feedback even more difficult. 

The current invention relates to a method and system for overcoming 
5 these limitations through development of a architecture which provides local 
intelligence and computation coupled with a method of compressing haptic 
information through various protocols which reduce the amount of 
information that is shared between the world modeler and the haptic 
system. System speed is improved and performance is enhanced making it 

10 feasible to employ multiple modes of haptic feedback simultaneously without 
overloading the communications link or basic computation engines. Since 
multiple modes of sensory feedback are utilized to reflect the real 
interactions between a human body part and the outside world, a realistic 
user interaction with a.virtual or remote environment is achieved. 

15 Therefore, an object of the invention is to provide local processing to 

each sense and display device so as to provide real-tune feedback to the 
human operator. 

Another object of die invention is to provide a modular framework in 
which to extend the sensory feedback capabilities as required by the human 
20 operator. 

Another object of the inventioii is to minimize the coinmunications 
bandwidth required between the haptic system and the world modeler. 

Another object of the invention is to provide an anthropomorphic 
system while retaining portability. 
25 Another object of the invention is to provide a system which can 

sense and/or display force, slip, vibration, shape, stiffness, friction, texture, 
pressure, ma$s, temperature, sound and deciromagnetic radiation to a 
human operator. 
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Additional objects and features of the invention will appear from the 
following description in which the preferred embodiments is set forth in 
detail in conjunction with the accompanying drawings. 

i 

5 Siimmary of the Invention 

A system for sensing and displaying sensory information to a human 
operator comprising a world modeler adapted to receive sensory input from 
the operator over a communicadons medium. The world modeler modifies 
either a virtual world which is a mathematical model of an environment, or 
10 modifies a real world which is a remote real environment. The world 
' modeler then sends sensory information from the real or virtual 
environments over the communication medium to the haptic system. The 
haptic system displays this sensory information to the human operator. The 
result is that the operator is able to manipulate a real or virtual environment 
15 remotely through the information sensed and displayed by the haptic system. 
The haptic system is composed of sensory modules. The sensory 
modules are further composed of a controller (i.e. CPU, DSP or 
microcontroller umt). communications unit, and sensory and/or display 
devices. 

20 The display devices can include heating elements to convey 

temperature, voice coils to coiivey vibration, goggles to display visible, 
motors to display force, brakes to display friction, speakers to convey sound 
and tactile sensors to convey various senses of touch. 

The sense devices can include encoders to convey position, strain 

25 gages for force, display deyice feedback to measure torque, thermocouples 
for temperature and cameras for vision. 

Sensoiy information is .compressed through various methods. One 
method is to transmit only those sensory cues required to convey the 
intended sense. Another method is to only select the sensory information 
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required for a task. Further, the data sfelected may be compressed further 
through conventional data compression techniques (i.e. JPEG for video). 

The cpiiiplete system may be scaled as needed. Mizmxial systems may 
only convey tactile information through a haptic h^d. Larger systems .may 
S utilize exoskeletons to ponvey joint information and force to the system. 

Brief Description of the Drawmgs 
The invention is described and explained in more detail below using 
the eimbodiments shown in the drawings. The described and drawn features, 
10 in other embodiments of the inventipn, can be used individually or in 
preferred combinations. The foregoing and other objects, features and 
advantages of the invention will be apparent from the following more 
particular description of a preferred embodiment of the invention, as 
illustrated in the accompanying drawings. 
15 FIGURE 1 is a block diagram of the present invention, 

FIGURE 2 is a block diagram of the world modeler of the invention. 
FIGURE 3 is a block diagram of the haptic system of the invention. 
FIGURE 4 is a block diagram of variation of sensory modules of the 
invention. 

20 FIGURE 5 is a block diagram of the invention as implenienting 

teleoperation. 

FIGURE 6 is a front view of the hand embodiment of the invention. 
FIGURE 7 is a side view of example tactile sensory used to 
convey slip. 

25 FIGURE 8 is a block diagram of the haptic system for the hand 

embodiment of the invention. 

FIGURE 9 is a flow chart of die control software for the hand 
embodiment of die invention. 

FIGURE 10 is a front view of the hand, arm and vision embodiment 
30 of the invention. 

6 
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Description of the Preferred Embodiment 
The current invention relates to a method of overcoming past design 
Jimitations through development of an architecture which provides local 
intelligence and computation in conjunction with compression of the haptic 

5 information through the use of protocols which reduce the amount of 
information needed to be shared between computation engines at a given 
instant. By utilizing local processing within the isensory units the system 
speed and performance is enhanced maldng it feasible to employ multiple 
modes of haptic feedback simultaneously without overloading the 

10 communications channel or basic computation engines. Multiple modes of 
feedback are necessary in order to have a realistic mteraction between a 
human body part and a virtual or remote environment. 

The present invention generally relates to a system architecture for 
providing sensory feedback in general to the human operator. In particular, 

15 touch feedback is necessary for interactions with the human operator in 

controlling either a computer generated, virtual environment or a real remote 
robotic enviromnent. 

SensQyy DispHay and Sense 

20 The sense of touch, or haptic sense, is complex. Unlike vision, touch 

involves a direct physical interaction, so that human actions cause and 
change what is perceived. It is the fundamental role of touch to sense the 
results of these contact interactions in order to guide manipulation and body 
motion. The current art only contains devices for presenting a crude tactile 

25 image to a location on the body. This type of information represents only a 
fraction of the total information sensed by the human sense of touch. Many 
different types of basic mfdrmation about the contacts are extracted by 
people through a combination of four different tactile sensory systems, 
measurement of joint motion, and measurement of joint torques . This basic 

30 information can be organized broadly into passively acquired mformatidn 

7 
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and actively acquired infpnnation. Passively acquired infpnnatioh can be 
. gathered witbout relative motion between the body and an object and without 
varying the forces between the body and the object. Actively acquired 
information requires either relative motion or varying contact forces. A 
5 summary of some of the basic types of cues or sensed infoxmatipn is given 
below. 

Passiveiv Acquired Identification Cues 

Passively acquired identification cues can be obtained by the act of 
10 holding the object in the hand. Three major cues have been identified in this 
category: 

1 . Detennining the local contact normal and curvature of the 
surface at each contact. 

Determining the local contact curvature and labeling it as a 

15 point, edge, or planar contact is important for grasp acquisition and 

object identification. Different contact types have different grasp 
properties which affect stability; Salisbury. J.K. and Mason, M.T.. 
Robot Hands and the Mechanics of Manipulation^ MIT Press, 
Cambridge, MA (1985). In addition, the local contact normal and 

20 curvature provide a stt-ong pruning heuristic rule for identifymg 

objects and object pose; Grmspuj W.E.L. , Object Recognition by 
Computer: The Role of Geometric Constraint ^ MIT Press, 
Cambridge, MA (1990). Local object curvamre can be measured 
passively be examining the normal contact force distribution. 

25 2. Determining the surface texture at each contact. Surface 

texmre affects girasp stability. Rough textures are generally easier to 
grasp than smooth texmres. In addition, surface texture can also be 
used as a pruning heuristic rule in identifying objects and object pose. 
Textore cues are produced both from the spatial distribution of the 
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contact force and the dynamic vibration effects produced at the skin 
during motion. 



5 3. Groiss object shape. 

By using the finger geometry and the location of the contact 
points in the hand, the shape of the object can be estimated. For 
example, an initial grasp might be a power grasp in order to get a lot 
of contact information about an object. Once the object and its shape 

10 is identified, a dexterous grasp might be used for manipulation. 

Actively Acquired Identification Cues 

With some local active exploration, properties relating the reaction 
forces to the applied force can also be determined: 
15 1. Determining the local stiffness at each contact. 

The local surface stiffness can be estimated by applying 
varying normal forces and measuring the change in contact area. 
Softer surfaces will produce a larger contact area for a given applied 
force. This is not a significant cue for maniptilation of control 
20 surfaces and tools since ahnost all of the objects irivolved are rigid. 

2. Determining local fiictional properties. 

The local ftiction along with the local contact type control the 
local grasp stability. Frictipn can be measured by applying varying 
tangential forces anSi tiien detecting the onset of slip (slip detection is 
25 a separate basic maiiipulation cue; see next Section). This task also 

teqmres the ability to measure tangential forces. 

3. Overall object stiffiiess. 

By integrating the contact force over an area with the joint 
torque's, and correlating this with joint deflection, the overall stiffhesis 
30 " can be computed.' 
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4. Detecting slip. 

Detecting the onset of slip between an Object and the hand is 
essential for grasp maintenance. Slip detection is used to determine 
the necessary grasp forces at each contact during all states of 
5 manipulation. 

5. Determiiung object mass (weight), center of mass, and 
moments of inertia. 

By manipulating a grasped object and measuring the rioting 
joint torque's and net contact forces at the hands for different 

10 configurations, the weight, center of mass and moments of inertia of 

the object can be computed. This information can be used for object 
identification and pose determination, Siegal, D.M., Pose 
Determination of a Grasped Object Using Limited Sensing y Ph.D. 
Thesis, Department of Electrical Engineering. MIT, Cambridge, MA 

15 (1991), as well as for computing the necessary torque's for throwing 

or manipulating the object. 
6; Estimating directions of contact constraint. 

Assembly is the process or bringing a moving part into a 
constrained relationship with a fixed part. In order to control the 

20 assembly process, the directions in which movement is constrained 

need to be estimated. Contract constraints are estimated using 
measurements of the reaction forces as a function of position and by 
measuring the direction of impact forces. 
7. Detecting changes in contact constraints . 

25 This is one of the most common tasks during inanipulation. 

The detents in switches, the termination in screws, die impacts from 
mating two parts are all examples. The onset of the change can be 
detected by looking for unexplained impact forces and the direction of 
the impact force. 

30 
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Current tactile feedback devices present a flat contact image using 
either arrays .of tactors or raster scan techniques. Appendix A includes a list 
of prior art devices whose teachings are herein incorporated by reference. 
This type of display can only be used for very local sivsp^ display ot 

5 textures, and is limited to a flat type of display. A need exists for devices 
for displaying a variety of the remaining cues, such as those described 
above, (slip, contact curvature, gross texture) and those cues such as 
vibration, for which devices already exist in a realistic and natural fashion. 
Thus the object of the current invention is to provide a modular strucmre 

10 under wMch individual haptic cues or multiple cues can bie effectively and 
efficiently presented to the human operator. 

Haptic Compression 

The bandwidth required to completely characterize and display haptic 

15 feedback of all senses would normally be prohibitive. Various forms of 
haptic compression are used to limit this bandwidth. 

Haptic compression is based upon human perception. The 
compression algorithms ask the question; what information has to be 
presented to perform this manipulation or recognition task. Haptic 

.20 compression is based in sending a set of physical parameters aiid a number 
indiciatiiig the model type to the haptic display computer. Locally, the 
model number is used to pull up a stored model of a component of the task 
physics from the memory of the local haptic processor. The parameters then 
control the specific instantation of that model. 

25 For example, perceiving a fine surface texture is done primarily 

through percepdon of spatial frequencies through vibration. By measuring 
relative velocity and having the spatial frequency stored as a two directions 
fourier spectrum the texture display would be able to simulate a complex 
texture. Compression occurs because (1) spatial representation does not 

30 resolve all surface features; (2) vibration is substituted for an ^ray display. 
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Curvature and surface normal is based on aii analysis of the task 
which shows that only this information is necessary for performing the 
required manipulation task. The coniputer sends 2D cufyature properties 
and 1 contact point to represent the surface to the haptic display computer. 
5 The display then takes care of bending up the surface. This is a first order 
representation of the more complex shape. Since not all the infomiation is 
shown, there is information compression. 

For local stiffness, the haptic processor gets a nmnber K which 
represents the contact stiffness. Based on this number it simulates a surface 
10 of stiffness K by applying the correct forces given a displacenient relative to 
the display . 

Like local stiffness, a slip haptic processor only needs the coefficient 
of friction niu. The slip processor can then measure the normal force and 
the tangential force to determine the current state of the slip feedback. 

15 In order to irnplemerit general constant constraints a set of constraint 

equations C_i(g,x) are stored on the local processor. The constraint 
parameters g and the specific model type i are sent to the local processor in 
order to instantiate a specific constraint model. The local process applies 
corrective forces to try and force the coordinates of the display, x, to obey 

20 this equation in the display . 

Information may also be compresised tiu-pugh the removal of 
extraneous information. One method is to include only those senses 
appropriate to the task. A second method is to use psychophysical 
compression, i.e. only including the sense cues necessary to give the 

25 operator the ability to "feel" the characteristic bemg displayed. A third 
method is to use data compriession prior to transmission across the 
communication chiaimel. A fourth method is data thinning , e.g. 
communicating only the changes since the last transmission or transmitting 
Only when changes are large enough to warrant. A fifth method is to 
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partition the system such that the computation aiid display subsystems share 
a lower bandwidth communication channel. 

Sense Selection 

5 There may be no need to include temperamre display , if the task is 

pne of simply grasping and moving an object pf known temperature. There 
rbay also be no need to transmit with absolute fidelity all hand joint torques 
if the application requires only the input of the thumb and forefinger for 
acceptable results. Selection of the nunimal number pf senses required for 

10 convincing haptic feedback is the primary method of reducing 
communications traffic. 

Psvchophvsical Considerations 

Studies have shown that the human may only need a subset of sense 
15 cues to synthesize the sensory feedback. Sense-mapping, or replacing one 
sense for another, can he used for feedback of senses or other data sources 
otherwise difficult to display. An example is using a vibrotactile display to 
display surface temperature through variations in vibration amplitude. 

20 Data Compression 

Various methods exist for brute-force data compression. The least 

efficient algoiithms, on a percentage compression basis, offer the ability to 

completelly and accurately reproduce the input data after decompression. 

More efficient methods, such as fractal compression, reproduce the data 
25 with reasonable fidelity and require relatively small transmission bandwidth 

but require some knowledge of the nature of the data beixig compressed. 

The best protocol will depend on the psychophysics of the senses being 

displayed. 

30 Data Thinning 

■ 13 
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Sending a continuous stream of absolute sensory data across the 
conununications channel may be inefficient. Once the minimuin bandwidth 
of the sense displayed is established, only state changes may need to be 
conununicated. The channel could become event driven by transmitting only 
5 perceptible changes. 

System Configuration 

The system may be partitioned so that there is a reasonable amount of 
computation performed by the display driving electronics. Each sense can 

10 have it's own processing, sharing and communicating a conmion knowledge 
of the "world" but reacting in ways unique to the sense being displayed. 
This approach is readily scalable due to its modularity. For example, the 
visual display hardware may be suited to fast graphics rendering but does 
not need the ability to compute other sense data. The only need is to be 

15 able to modify and display its world view given input data from other 
transducers and senses. 

The prototypical haptic module would consist of a processor, 
memory, power and drive electronics, transducer interfaces, and a 
communications interface and controller. The software on board the haptic 

20 processor would include the kinematics mapping the sensor data to the 
display. If, for example, finger force feedback was desired, the module 
would query the fmger's position sensors, map this data to joint angles and 
finger geometry in it's world view, seiise collisions with objects iii it's world 
view^ and calculate resultant forces. It would communicate the updated 

25 world view to the other modules and apply the appropriate torques to the 

finger, knowing the configuration of the force feedback device. The system 
is tolerant of system communication latencies as the feedback loop is local to 
the haptic processor. 

30 Scaling and Mapping Sensory Information 
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With the sensors and transducers available, the option of mapping one 
sense to another becomes feasible and perhaps desirable in some cases. The 
ability to map physical parameters not normally sensed (for exiample 
electromagnetic field strengths) to human senses is compelling and may even 

5 lead to more intuitive understanding of complicated phenomenon. Sense 
data may be freely scaled, such that a remote robot manipulator could 
experience forces on the order of tons that could scale to ounces on the 
haptic display. At the other end of the spectrum, microfpiices can be scaled 
to pounds at the haptic display, allowing naicromanipulatioh with exU'emely 

10 fine resolution of control. 

General System Architecture 

Referring now to the drawings, wherein like numerals are used 
throughout the several views to identify like elements of the invention, there 

15 is disclosed a system for remote multimode sense and display as shown in 
Fig. l. The haptic system 14 interacts with a user 16. Sensory input 19 
from the user 16 is collected by the haptic system 14. This information is 
compressed by the haptic system 14 and transmitted onto the 
conununications link 12 to the world modeler 10. The world modeler 10 

20 uses the information to manipulate either a real or virtual world. The world 
modeler monitors and measures the state of the world, transmitting display 
information onto communications link 12 to haptic system 14, where the 
display mfprmation is converted to sensory ouq>ut 18 for the user 16. 

With reference to Fig. 2, the world modeler 10 of Fig. 1 may take 

25 two fonns 10a or 10b. The first form 10a utilizes a computer system 20 to 
create a mathematicai model 22 of a virtual world. The second form of 
world modeler 10b utilizes a cpinputer system 24 to sense information 25 
and send control information 26 to remote input-output system 28. This 
remote input-ou^ut system 28 exists in a real envkonment. 
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The haptie system 14 of Fig. 1 is further composed of at least one 
sensory module 30 as shown in Fig. 3. Each sensory module 30 may take a 
variety of formats. Fig. 4 shows examples pf sensory modules 30a, 30b and 
30c. A sensory module is composed at a cbntrqller 40a communications 

5 unit 44 and at least one sense unit 42 or display unit 43. Sensory module 
30a contains only one sense unit 42, sensory module 30b contams only one 
display unit 43, while sensory module 30c contains two sense modules 42 
and one display unit 43. 

Sense unit 42 contains sensors to ihonilor sensory input from the user. 

10 For example, this sensory input may be force, position, pressure, 

temperature information as experienced by the user. Display unit 43 
contains displays to convey infonnation, such as force, temperature, 
vibration, texture, friction, shape and stif&ess, to the user. The result is the 
user interacts with either a remote or virtual world through sensory input 

15 controlled by the haptic system. 

Fig. 5 shows the system implementing teleoperation. A user 56 
interacts with the remote world 50. The haptie system 14 gathers sensory 
input 52, This information is compressed by the haptic system 14 and 
transmitted on conununications unit 12 to the world modeler 10. The world 

.20 modeler generates control information 26 from the conipressed sensory input 
received from the communications link 12. The remote input-output unit 28, 
which may be a robot or mechanical arm, responds to the control 
information 26, thereby acting in the remote environment 50. The world 
modeler also takes sensory input 25 sensed by the feniote input-output unit 

25 28 from the remote environment 50. this sensory input 25 is cpnipress^ 
by the world modeler 10 and transndtted onto communications link 12 to the 
faapdc system 14. There the compressed sensory infonnation is displayed 54 
to the user 56. The result is that user 56 controls a remote input-output 
device 28 in a remote world 50. 

30 
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Hand Haptic System 

The hand haptic system is comprised of touch sensory modules which 
are composed of a local processing element connected to display and sense 
hardware via wire or fiber optic connection. The sensory module jis in turn 
5 connected via a wire or fiber optic link or a communications link such as an 
ethernet port, serial port, bus connector such as a Bit 3, parallel port, game 
port, a local area network to the world model transmitting either direct 
information for simple or low speed applications or information which has 
undergone haptic information compression. The human operator controls 
10 the world model through interactions with the visual model information 
(displayed on a computer monitor) and the touch display hardware. 

This results in the touch display hardware reacting and displaymjg 
touch information to the operator. The touch sensory module, because it has 
a local processor system, reacts directly to the human stimulus and thus can 
15 interact at the high speeds required for the feedback to feel realistic even in 
the face of rapidly changing stimuli from die human operator. This reaction 
caii be completely independent from the world model and visual display and 
thus not be slowed by a slow communications link and can therefore run at 
speeds upwards of IKHz if necessary. This is made possible because of the 
20 architecture of the local processor unit. This processor can be a single chip 
such as a DSP or Microcontroller, a board with multiple chips or a 
computer, depending upon the number of sensor and display elements 
contained in the touch display hardware and the complexity of the model. 

The haptic compression is achieved by utilizing the basic research into 
25 human perception of touch described above to determine which touch cues 
are bemg utilized for a given type of manipulation. For example, when m 
object is grasped the jpint torque cues as well as slip information are used to 
ensure that the object is stably grasped. The algorithms in the local 
processor detennine the type of cues required and their magnimde thus 
30 reducing the amount of information which needs to be transmitted to the 
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Other system components. The world model only needs to know the 
resulting object status (i.e. position and orientation, or forces acting on the 
object, depending upon how the world model is structured) thus the 
information content of the transmitted sigiials is greatly reduced thus 
5 improving the overall perfoiinance of the system. 

In the embodiment shown in Fig. 6, three sense and display elements 
were used; slip on the finger tips, force feedback to the index finger and 
thumb, and vertical resistance. In addition the device provides position and 
orientation of each of the system elements. 

10 In this embodiment a simple slip display, showii in Fig. 7, was 

developed as disclosed in U.S.S.N. 08/187,646. entitled "Multimode 
Feedback Display Technology " filed January 27, 1994 (herein incorporated 
by reference) . The slip display consists of a debrin cylinder 71 with a 
textured surface, directly mounted on the shaft of a miniature gear motor 72. 

15 The cylinder is supported on the other side with a miniature bearing 73, 
The motor is held by a set screw in the mounting bracket 70. 

In Fig. 6, the slip display 61 was attached to the tip of each of the 
index finger force feedback apparatus 62 and the thumb force feedback 
apparams 63. Force reflecdon to the index finger is provided by a modified 

20 two-degree-of-freedom S AFIRE (Sensing and Force reflecting Exoskeleton, 
Patent application serial #07/961,259 submitted 10/15/92) prototype, which 
consists of two cable-driven linkages 65, with the two niotors 66 mounted on 
the forearm. The axes of rotation of the linkages are co-located with the 
PIP and MCP joints of the* index fmger. Force reflection to the thumb is 

25 provided by a one-degree-of-freedom device 67, also driven by cables 69 
from a remote motor mounted on the forearm 68. 

The vertical resistance and position aiid orientation in space is 
provided by a boom 64. The vertical force reflection axis provided up to 2 
lbs of virtual weight at the hand. It also supported the weight of the 

30 S AFIRE through counterbalancing. The stif&iess of the vertical boom up to 
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the SAFIRE attachment bracket exceeded 140 lb/in, the minHnum stiffness 
required for a cantilevered beam to feel rigid to a human operator. The 
resolution of the vertical and horizontal position sensors for the hand and 
index finger was 0.005 inch. Two linear ways were mounted on opposite 
sides of the vertical support structure, which was rigidly attached to a heavy 
base. 

The drive mechanism 201 for the vertical dejgree-of-freedom was 
located on the back of the support smicture 200. The motor providing 
verdcal force reflection was horizontadly mounted on the slider riding in the 
rear linear way. A threaded capstan was directly mounted on the shaft of 
the motor. A 7x7, O.032 inch diameter steel cable was wound 3-4 turns 
around the capstan. One end pf the cable was terminated on a cable 
tensioner at the top of the structure. As the motor rotated, the cable hoisted 
the slider up and down the rear linear way. 

The rear slider was connected through a second cable and a set of 
pulleys to a plate mounted on two sliders riding in a second linear way, 
mounted on the front side of the vertical support structure. Using two 
sliders instead of one greatly improved the twisting and bending load 
capacity of the front linear way . The two sliders were mounted on the plate 
through two pivots to minimize bending moments applied to the front linear 
way. 

Vertical forces were transmitted to the SAFIRE by a horizontally 
mounted linkage. The linkage consisted of a distal block attached to the 
front slider plate, proximal'blpck, and a 12 inch long rod. The distal block 
housed the bearings and the encoder for the first passive pivot, also called 
the distal pivot. One end of the rod was pivoted about this block so that die 
linkage rotated freely in the horizontal plane. The other end of the rod was 
pivoted about a second bearing/encoder housing, also called the proximal 
pivot. The proximal mounting block was attached to the SAFIRE base with 
a rigid bracket. The length of this bracket was designed to position the hand 
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SO that the proximal pivot was between the tip of the index finger and the 
thunlb. 

The motor for vertical force reflection was selected based on the 
power requirements. Ideally the motor should produce a force of 2 lbs at 
5 the hand. It is estimated the band could move at a maximum vertical ^peed 
of 8 in per sec. We selected a 90W motor without a gearhead to provide a 
smooth operation with mininial friction. This is designed to produce .a force 
that exceeds the corresponding frictional force, if the frictional coefficient 
between the finger and the display is estimated at 0.5, the tactile display 

10 must be able to produce 1 lb. of force. If the display moved past the finger 
at 1 in/s, the required power would be around 0.112W. A motor producing 
0.25-0. 5 W would therefore be a suitable actuator for this display. The 
MicroMo Series 1016 motor fits these requirements; it is 0.384 inch in 
diameter, 0.63 inch long and weighs 0.23 oz. This is mounted in the space 

15 beyond the fingertip. 

The hand haptic system is interfaced to the world modeler, in this 
case implemented on a graphics rendering PC, through a local processor 80, 
in this case a VME card cage which houses a TI C30 DSP board 81, I/O 
boards 82, 83, and 84, and a BiT3 VME-ISA bus adapter 85. 

20 The physical siniulation and device drivers reside on a TI C30 

floating point DSP board :81. The DSP board is installed m a multislpt 
VME cage which also houseis the I/O electronics. The simulation is 
interrupt, driven with a time step of 300 Hz, which is the minimum 
performance level necessary for a stable and realistic simulation. Higher 

25 performance could be gained by parsing out the simulation tasks among 
several DSP processors and executing them in parallel. For example, 
commmiication tasks widi the graphics PC and boom position sensing could 
be handled by a different processor than the main simulation loop. 

The DSP simulation, as shown in Fig. 9, communicates with the 

30 graphics PC via a Bit3 VME-ISA bus adapter. This high speed 
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coitunumcations channel handles graphics information generated by the 
simulation and simulation parameters generated by the user interface. The 
user interface starts the standard simulation demonstration and provides 
projgrara status information. Since the user interface and graphics rendering 

S . can be somewhat asynchronoios to the simulation, graphics and PC system 
latencies will not dismrb the simulation which depends oii a uniform time 
step. Also the bandwidth required for this information is low enough to not 
require a dedicated processor. This ;Software approach does not require the 
simulation and graphics to conunuriicate a significant amount of objects 

10 characteristics upon startup of the simulation so that a minimum of data is 
exchanged during run time. Data from the isimulation need not be 
transmitted to the graphics PC any faster than the graphics refresh rate. For 
example, at a 60 Hz refresh rate, new data would be needed at a maximum 
of every third integration time step. 

1:5 Sensing the joint .positions and orientations and applying appropriate 

forces and slip requires additional I/O electronics. The joint positions are 
sensed with digital position encoders. The joint torque's and slip applied to 
the touch display are voltage controlled by the simulation via 6 12-bit digital 
to analog converters and external current mode motor amplifiers. The 

20 encoder counters and digital to analog converters reside on the high-speed 
VME bus with the DSP board and are directiy accessed by the DSP. The 
generality of the control electronics and overall architecture makes the 
system adaptable to other fprce reflecting masters. The minimum 
configuration can handle six output and eight input degrees of freedom, and 

25 can be easily expanded to any reasonable number by adding I/O modules 
iand, if necessary, DSP processors. 

Almost all of the application grapihics and DSP code is written in C. 
The simulation can be developed on die graphics PC and debugged with 
tools available for the PC/BiT3 development environment. Portmg the 

30 simulation to the DSP i$ then relatively straightforward. In fact, die existing 
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simulation can be debugged on the PC and then recompiled for the DSP with 
only minor changes. Hardware ba$ed debuggir^g tools are available for the 
DSP, although the PSP/PC interface provides a relatively powerfiil 
stethoscope for all but the most low level debuggmg or obscure 
5 compiler/hardware bugs. The DSP executable and source code resides on 
the PC and is downloaded to the DSP at run time. 

This systena and method can be applied to a .wide variety of physical 
embodiments including simpler systems and more complex and capable 
systems with more feedback modes utilized. 

10 

Hand/Arm/Vision Haotic System 

An alternate embodiment utilizes the hand haptic isystem in 
cdnjunctipn >yith an exoskeletail arm mechanism combined with a vision 
display. The exoskeleton arm consists of two interconnected mechanisms, 
15 one for the shoulder and the other for the elbpw and forearm as shown in 
Figure 10. 

The shoulder mechanism 90 consists of a three DOF motor driven 
gimbal, a paissive prismatic joint (spline), a 2 DOF passive gimbal and a 
upper arm cuff. The elbow mechanism 92 is connected to the shoulder 

20 mechaniism 90 at the distal end of the spline 91 . This allows the system to 
accoi^modate changes in the distance between the shoulder 90 and elbow 
mechanisms 92. The spline also elimiiiates the need for adjustinent for 
different upper arm tengths. 

The center of the distal gimbal 93 is located oh the axis of the upper 

25 arm E/I rotation. The elbow F/E motor 94 is pivotally jointed to both the 
upper 95 and lower 96 cuffs. The final step was to use a brake 97 at the 
distal gimbal pf the shoulder mechani$m to ground the elbow/fpreann 
mechanism. 
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The elbow mechanism has two DOF. The first DOF is aligned with 
the axis of elbow F/E. The second DOF is placed along the forearm with a 
remote center drive mechanism. 

The three motors 100, 101, and 102. at the shoulder gimbal can be 

5 coordinated to apply a force at the upper arm which is approxitoately 

perpendicular to and intersects with the longitudinal axis of the upper arm. 
TUs force creates a torque about axes of the shoulder F/E and Ab/Ad. The 
motor at the elbow controls the Flexion/Extension torque about the elbow. 
Tlie rotational torque about the longitudinal axes of the upper and lower 

10 arms are controlled in two different ways depending on the position of the 
elbow joint. When the elbow is extended the motor at the wrist applies 
rotational torque to the longitudinal aces of both the upper and lower sunns, 
since their motions coincide. When the elbow is flexed, the motors at the 
shoulder can be coordinated to apply a force at the forearm cuff to produce 

15 a torque about the upper arm E/I, and the RCD applies a torque to the 
forearm. 

The biggest advantage of this design was that it conformed to the 
anatomic motions of the entire arm. It imposed no kinematic constraints on 
the arm because the shoulder mechanism had full 6 DOF and the elbow F/E 

20 mechMiism formed a 4-bar linkage with the elbow F/E joint. Therefore, it 
was very comfortable to wear. It also offered large Range Of Motions 
(ROM). It not only applies forces to the entire arm grounded to the back, 
but also torque to individual joints. The former allowed us to simulate 
picking up an object by bracing between the forearm and the upper aim or 

25 between the entire arm and the chest. The latter could smiulate contacts of 
objects in a virtual ejxvironimeint by any portion of the arm such as hand, 
forearm and upper arm. In smrunary, this mechanism offered all the force 
display requirements for a force reflective arm master and conformed to the 
aiiatomic motion of the arm. 
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A higher performance version can be achieved by using magnetic 
brakes in conjunction to tow miniature motors or actuators. The two 
magnetic particle brakes may be used to replace the fost two motors at the 
shoulder gimbal. In order to use brak^, the mechanism must be fiiliy 
5 balanced about these two axes becaiuse brakes are passive and cannot lift. In 
addition, the miniature actuators or motors must be put in series with, and 
parallel to, the two brakes for active force control. One way to achieve this 
is to activate the 2 DQF passive gimbal at the upper arm cuff with such 
miniature motors. With the brakes locked, the two miniature actuators or 

10 motors can display small forces to Ae operator at very fine resolutions. To 
simulate a haiid contact, simply lock the brakes and drive the two miniature 
motors to their respective hard stops so that the braking forces can be felt by 
the tipper arm. For a force in between these two extremes, the combination 
of Pulse Width Modulations (PWM) on the brakes and force controls on the 

15 miniature motors can be used simultaneously. 

The video sense is accommodated via a head mounted display or 
remote video screen, with accompanying processing. They can 
communicate with each haptic processor, which take in data from the 
physical parameters of the sense being displayed, e.g. position of the fingers 

20 related to force feedback. Each haptic processor applies the appropriate 
feedback, and communicates its part of the world view to the other 
processors. The communication channels can be fibre optic, infrared, 
wired, or whatever medium is appropriate to the twin demands of bandwidth 
and portability. Adding another sense is as straightforward as adding 

25 another sense module up to the limitations of intef-module bandwidth. 
System autoconfiguration is a straightforward fiuiction of the software| 
Controllers for the modules could he general purpose digital signal 
processors for maximum flexibility or specialized processors, e.g! 
application specific integrated circuits, for niinimum cost. 
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Vision sense information is provided to the opierator through a set of 
goggles containing vision displays . Such a device is the Vision Immersion 
'Module manufactured by Kaiser Electro-Optics in Carlsbad, CA 92008. 
Sound may sdso be provided to the operator through a set of headsets. 

5 

Portable Svstein 

The complete systenoi is worn by the human operator through a 

baclq>ack mechamsm which also contains the electronic and power systems. 

The resulting system is a self-supporting exoskeleion, requiring no precise 
10 alignment to conform to the anatomical movement of the ann, and is thus 
. comfortable to wear. The device is comfortable to use because it applies 

normal forces co the arm; no twist, no shear to skin or bones. It is also easy 

to don and doff because entire mechanism is supported through the 

backpack. 

15 The complete system communicates to the world modeler through a 

cable which also provides power to the system. The system can also be self 
contained by providing power through batteries and communications through 
an RF or m link. 

The techniques of haptic compression, combined with modern power 

20 electronics and digital signal processing, make it possible to construct a 

totally portable haptic display systeni. . Such a system can be used to explore 
virtual worlds in architectural applications, control remote mobile robots in, 
space or nuclear applications, or transverse any number or databases to 
explore them through haptic feedback. 

25 While the invention has been particularly $hown and described with 

reference to the preferred embodiments thereof, it will be understood by 
those skilled in the art that various changes in form and detail may be made 
therein without departing from the spirit and scope of fee invention. 
What is claimed is: 
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1 . A system for sensing and displaying sensory infonnation to a user 
comprising: 

an world modeler adapted to receive first sensory informatipn and 
transmit second sensory information, 
5 a communications mediimi adapted to carry the first seiisory 

informatipn and the second sensory information, 

a hapdc system adapted to sense the first sensory infQrmatipn and 
transmit the first sensory information on the communications medium to tiie 
environment creator, adapted to receive the second sensoiy information 
10 transmitted on the cpmmimications medium by flie environment creator and 
further adapted to display the second sensory infonnation, 

whereby the system senses sensory information from the user and 
displays sensory information to the user. 

15 2. The system of claim I wherein the world modeler is a computet 

system, 

the computer system adapted to create a matheniatical model of an 
environment, 

the computer system farther adapted to ncianipulate the matheniatical 
20 model of an environment based upon the received first sensory information, 
the computer system further adapted to create the second sensory 
information based upon tiie mathematical model of an environment and the 
first sensory information, 

whereby the computer system creates an iartificial environment for the 
25 user to interact. 

3 . The system of claim 1 wherein the world modeler is a computer 
system and a remote input-output device, 

the remote input device adapted to sense third sensory infonnation 
30 from the remote input-output device, 

26 



wo 95/20788 



PCtAJS9S/01227 



the computer system adapted to manipulate the third sensory 
information to create the second sensory information, 

the computer system further adapted manipulate the first sensory 
information to create fourth sensory information, 
5 the remote input-ouqput adapted to respond to the fourfli sensory 

information, 

whereby the remote input-output device is interactively controlled 
through the computer system by the user. 

10 4, The system of claim 1, 2 or 3 wherein the haptip system 

comprises: 

at least one sensory module adapted to: 
sense direct sensory information, 

process the direct sensory information into the first sensory 
. 15 information, and 

transmit the first sensory information, 
whereby the sensory module communicates the direct sensory 
information processed by the sensory module to die world modeler. 

20 5. The system of claim 4 wherein the sensory module is further 

adapted to: 

. receive the second sensory information, 
process the second sensory information into direct dispilay 
. information, and 
25 display the direct display infonnation, 

whereby the sensory module displays the direct display information 
processed by the sensory module to the user . 
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